Python Build System#302
Conversation
182e8a5 to
4aa0abb
Compare
|
Also may close #127 |
1ea2b13 to
db39910
Compare
4d78860 to
4288703
Compare
|
@CavRiley amazing effort!! 🥇 🔥 With a cursory review, I noted that we do not need to support manylinux2014 for ITK 6. A few preliminary questions (I have not taken a detailed look):
See https://www.perplexity.ai/search/we-have-a-python-package-itk-t-MLAY4xnZRCeU9SGHhyJHQQ |
|
Thank you, @thewtex!
Okay, I will remove this!
This does not use pixi-build, but I think it would be possible to use pixi-build to build the remote modules if not the base ITK modules as well. However, I would need to investigate this further to confirm it can handle ITK's CMake/SWIG wrapping.
Possibly, but I think the itk C++ conda-forge package would need to be built with the python wrapping (I'm not sure if this is currently the case) in order for it to be re-used here.
The |
4288703 to
86b1e2f
Compare
manylinux2014 is not needed for ITK 6. Remove the manylinux2014 image tag resolution, the manylinux1/manylinux2014 pixi-to-pattern renaming entries, and update shell script comments to reference _2_28 and _2_34 as the supported manylinux versions. Addresses review feedback from @thewtex on PR #302. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
manylinux2014 is not needed for ITK 6. Remove the manylinux2014 image tag resolution, the manylinux1/manylinux2014 pixi-to-pattern renaming entries, and update shell script comments to reference _2_28 and _2_34 as the supported manylinux versions. Addresses review feedback from @thewtex on PR InsightSoftwareConsortium#302. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
afed60a to
1af5a71
Compare
Add pre-commit hooks in Utilities/Hooks/ matching ITK's directory structure and commit message enforcement: - prepare-commit-msg: Inserts ITK prefix instructions into the commit message editor so developers see valid prefixes while composing. - kw-commit-msg.py: Validates commit messages with the same rules as upstream ITK: prefix check (BUG/COMP/DOC/ENH/PERF/STYLE/WIP), subject line max 78 chars, no leading/trailing whitespace, empty second line, and Merge/Revert exemptions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…n scripts Replaces the primary build orchestration layer with a Python-based system. Adds core abstractions including BuildManager for step-wise build persistence/resumption, CMakeArgumentBuilder for managing CMake definitions, cross-platform venv utilities, subprocess helpers, and pixi environment integration. Establishes platform-specific base classes that Linux, macOS, and Windows implementations extend. Restructures CMake layout into modular BuildWheelsSupport and SuperbuildSupport components. Shell scripts remain where still required. Co-authored-by: Hans J. Johnson <hans-johnson@uiowa.edu>
… Windows Implements platform build scripts using the new Python infrastructure. Linux/manylinux builds use dockcross with auditwheel for manylinux compliance. macOS builds integrate delocate and respect MACOSX_DEPLOYMENT_TARGET. Windows builds use delvewheel for wheel repair. Removes legacy shell scripts superseded by the new Python implementations. Co-authored-by: Hans J. Johnson <hans-johnson@uiowa.edu>
Adds make_tarballs.sh and integrates tarball generation logic into build_python_instance_base.py. Supports creating, downloading, and reusing ITK build caches across Linux, macOS, and Windows, including manylinux tarball support, arch-specific naming, and a --build-itk-tarball-cache CLI option. Adds make_windows_zip.ps1 for streamlined Windows wheel packaging and zip creation.
Removes unused and outdated shell scripts and legacy build utilities superseded by the new Python build system. Adds pre-commit hooks with linting applied across the codebase. Drops Python 3.9 support as it is EOL. Includes README and documentation updates for the new build workflow.
Add a rattler-build recipe to produce a conda package containing ITK C++ libraries with full Python wrapping artifacts (SWIG interfaces, CastXML outputs, compiled Python modules, stubs). This package can be consumed as a host-dependency to skip the expensive ITK C++ build when generating Python wheels for ITK or remote modules. The recipe supports overriding the ITK source repository and tag via ITK_GIT_URL and ITK_GIT_TAG environment variables, enabling builds from feature branches and PRs for testing. Build scripts are provided for both Unix and Windows platforms. CMake flags match conda-forge's libitk-feedstock (using system libraries) plus the additional wrapping options needed for Python wheel generation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a pre-built ITK is available in the conda/pixi environment (via the libitk-wrapping package), automatically skip the superbuild and C++ compilation steps and point the wheel builder at the installed ITK CMake config directory. Detection checks CONDA_PREFIX and PIXI_ENVIRONMENT_DIR for an ITK CMake config (lib/cmake/ITK-*/ITKConfig.cmake). When found, steps 01_superbuild_support_components and 02_build_wrapped_itk_cplusplus are replaced with no-op stubs, reducing remote module build times from ~1-2 hours to ~15 minutes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update the Linux, macOS, and Windows download-cache-and-build scripts to detect a pre-installed ITK from a conda/pixi environment before attempting to download ITKPythonBuilds tarballs. When ITKConfig.cmake is found under CONDA_PREFIX or PIXI_ENVIRONMENT_DIR, the entire download/extract phase is skipped and the build proceeds directly using the conda-provided ITK. This is backward-compatible: when no conda ITK is detected, the scripts behave exactly as before (tarball download path). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add commitizen configuration to pyproject.toml that enforces the ITK commit message convention (PREFIX: Description) with interactive prompts for selecting the change type. Valid prefixes: BUG, COMP, DOC, ENH, PERF, STYLE. The commitizen pre-commit hook in .pre-commit-config.yaml validates commit messages at the commit-msg stage. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add conda-forge-ready recipe files (meta.yaml, build.sh, bld.bat) for the libitk-wrapping package. These are prepared for submission to conda-forge/staged-recipes once the ITK install rule changes are merged upstream and an ITK 6 release is tagged. The recipe builds ITK C++ with Python wrapping using conda-forge system libraries and includes the CMAKE_FIND_ROOT_PATH settings required by conda-build cross-compilation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document the libitk-wrapping conda package integration, conda ITK auto-detection behavior, and the ITK commit message convention enforced via commitizen. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document the libitk-wrapping conda package workflow for building remote module wheels without recompiling ITK from source. Includes instructions for building and installing the package, environment variable overrides for custom ITK branches, and cross-references between the ITK and module build docs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remote modules hard-code ITK sub-package version pins in their pyproject.toml (e.g., itk-io == 5.4.*). When building against ITK 6, these pins cause pip install conflicts. Add _update_module_itk_deps() which rewrites exact ITK pins to minimum version constraints (>= 5.4) before invoking scikit-build-core. This runs automatically during build_external_module_python_wheel() when ITK_PACKAGE_VERSION is set, so remote module wheels always have dependency metadata compatible with the ITK version they were built against. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Back up pyproject.toml to pyproject.toml.orig before rewriting ITK dependency pins, then restore the original after the wheel is built. The modified version is saved as pyproject.toml.whl for reference. Uses try/finally to guarantee restore even if the build fails. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document the plan to migrate from build-time pyproject.toml rewriting (Strategy 1) to a scikit-build-core dynamic metadata provider (Strategy 3) that resolves ITK dependency versions at build time without modifying source files. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a remote module declares dynamic = ["dependencies"] in its [project] table, skip the pyproject.toml rewrite (Strategy 1) and instead set ITK_PACKAGE_VERSION in the environment for the scikit-build-core metadata provider to pick up (Strategy 3). Modules that have not yet opted into dynamic dependencies continue to use the build-time rewrite fallback, so both approaches coexist during the transition. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Limit the build-time pyproject.toml rewrite to the six ITK base sub-packages (itk-core, itk-numerics, itk-io, itk-filtering, itk-registration, itk-segmentation) whose versions are tied to the ITK release. Remote module cross-deps like itk-meshtopolydata are versioned independently and are not rewritten — a warning is printed instead so they can be reviewed manually. Prevents incorrect rewriting of e.g. itk-meshtopolydata == 0.12.* in BSplineGradient and WebAssemblyInterface. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The dependency floor must match the ITK series the wheel was compiled against, not a hardcoded backward-compatible value. For ITK 6.0.0b2 the rewrite now produces "itk-io >= 6.0" instead of "itk-io >= 5.4". This ensures wheels are only installable alongside the correct ITK release series. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remote modules must support installation with any ITK from v5.4.0 through the latest release. Use a fixed floor of >= 5.4 rather than pinning to the version being built, so wheels are installable across the full supported range. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
A wheel compiled against ITK 6.0 must require >= 6.0 at install time, not >= 5.4. The floor reflects the ITK series the wheel was linked against. The build system supports building against any ITK from v5.4 through the latest — the floor is derived dynamically from ITK_PACKAGE_VERSION at build time. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
03cc291 to
221359d
Compare
…main Add shell and Python scripts that clone ITK, ITKPythonPackage, and all remote modules from their latest main branches, then build ITK Python wheels followed by every remote module that has Python wrapping. All wheels are collected into a single /tmp/<timestamp>_LatestITKPython/dist directory. Both scripts parse ITK's Modules/Remote/*.remote.cmake files to discover remote module git repositories, filter to those with wrapping/ and pyproject.toml, and build each module wheel reusing the ITK build from the first step. Usage: ./scripts/build-all-latest-wheels.sh [platform-env] python scripts/build_all_latest_wheels.py --platform-env linux-py311 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add --itk-ref option to both shell and Python build-all-latest-wheels scripts. Defaults to "main" but accepts any git ref (branch name, tag like v6.0b02, or commit hash). The ITK clone is now a full clone (not shallow) to support arbitrary refs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Python-based build system for generating Python wheels
Supersedes @hjmjohnson draft PR #299
This refactoring transitions build logic entirely to Python scripts while maintaining the shell script entry points for cached builds either locally or in the GitHub Action environment. A couple of the build improvements are outlined below:
Some developer improvements:
Migration Guide for GitHub Actions
To migrate the GitHub actions work flows to use this new build infrastructure, new caches builds need to be created with this build system (build paths and names changed to be consistent across environment).
Once the cache files are uploaded to the ITKPythonBuilds repository the actions can use the specific Git tag the caches are uploaded under. To use this infrastructure you would also need to specify the branch this addition would go under. For example:
Backwards compatibility
I'd appreciate any feedback you could provide regarding improvements or issues